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© A scene recognition system and a method em- 
ploying low and high level feature processing are 
disclosed. Image data is processed by a low level 
feature detection processor (31) that extracts low 
level features from an image. This is accomplished 
by converting a matrix of image data into a matrix of 
orthogonal icons that symbolically represent the im- 
age scene using a predetermined set of attributes. 
The orthogonal icons serve as the basis of process- 
ing by means of a high level graph matching proces- 
sor (37) which employs symbolic scene segmenta- 
tion, description, and recognition processing that is 
performed subsequent to the low level feature detec- 
tion. This processing generates attributed graphs 



representative of target objects present in the image 
scene. High level graph matching compares pre- 
determined attributed reference graphs (36) to the 
sensed graphs (35) to produce a best common sub- 
graph between the two (35, 36) based on the degree 
of similarity between the two graphs (35, 36). The 
high level graph matching (37) generates a recogni- 
tion decision based on the value of the degree of 
similarity and a predetermined threshold. The output 
of the high level graph matching (37) provides data 
from which a target aimpoint is determined, and this 
aimpoint is coupled as an input to a missile guid- 
ance system (24) that tracks identified targets. 
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CROSS REFERENCE TO RELATED APPLICA- 
TIONS 

The present application is related to European 
patent applications Serial No. 91 105 251 filed on 
April 3, 1991, entitled "Improved Data Compression 
System and Method", and Serial No. 91 105 250 
filed on April 3, 1991, entitled "Improved Data 
Decompression System and Method", whose 
teachings are incorporated herein by reference. 

BACKGROUND 

The present invention relates generally to 
scene recognition systems and methods, and more 
particularly, to a scene recognition system and 
method that employs low and high level feature 
detection to identify and track targets. 

Modem missile scene recognition systems em- 
ploy specialized signal processing architectures 
and algorithms that are designed to quickly and 
efficiently detect the presence of particular target 
objects such as buildings, trucks, tanks, and ships, 
and the like that are located in the field of view of 
the missile. Consequently, more sophisticated de- 
signs are always in demand that can accurately 
identify or classify targets in a very short period of 
time. 

SUMMARY OF THE INVENTION 

The present invention comprises a scene rec- 
ognition system and method for use with a missile 
guidance and tracking system, that employs low 
and high level feature detection to identify and 
track targets. The system employs any conven- 
tional imaging sensor, such as an infrared sensor, a 
millimeter wave or synthetic aperture radar, or so- 
nar, for example, to image the scene. The output of 
the sensor (the image) is processed by a low level 
feature detection processor that extracts low level 
features from the image. This is accomplished by 
converting a matrix of sensor data (image data) into 
a matrix of orthogonal icons that symbolically re- 
present the image using a predetermined set of 
attributes. The orthogonal icons serve as the basis 
for processing by means of a high level graph 
matching processor which employs symbolic scene 
segmentation, description, and recognition process- 
ing that is performed subsequent to the low level 
feature detection. The process generates attributed 
graphs representative of target objects present in 
the image. The high level graph matching proces- 
sor compares predetermined attributed reference 
graphs to the sensed graphs to produce a best 
common subgraph between the two, based on the 
degree of similarity between the two graphs. The 
high level graph matching processor generates a 



recognition decision based on the value of the 
degree of similarity and a predetermined threshold. 
The output of the high level graph matching pro- 
cessor provides data from which an aimpoint is 
5 determined. The aimpoint is coupled as an input to 
the missile guidance system that tracks an iden- 
tified target. 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 

The various features and advantages of the 
present invention may be more readily understood 
with reference to the following detailed description 
taken in conjunction with the accompanying draw- 
75 ings, wherein like reference numerals designate 
like structural elements, and in which: 

Fig. 1 is a block diagram of a scene recognition 
system employing low and high level feature 
detection to identify and track targets in accor- 
20 dance with the principles of the present inven- 
tion; 

Fig. 2 is a detailed functional block diagram of 
the system of Fig. 1 ; 

Figs. 3a-3f show the processing performed to 
25 achieve low level feature detection; 

Figs. 4a and 4b show the results of flat linking 
and region growth processing performed in the 
scene segmentation section of Fig. 2; 
Figs. 5a-5c show the results of boundary forma- 
30 tion and linear feature extraction processing per- 
formed in the scene segmentation section of 
Fig. 2; 

Figs. 6 and 7 show attributes computed for 
region and ribbon objects, respectively, in object 

35 formation processing performed in the scene 
segmentation section of Fig. 2; 
Figs. 8a-8c show the transitional steps in gen- 
erating an attributed sensed graph; and 
Figs. 9a and 9b show a reference scene and an 

40 attributed reference graph, respectively; 

Fig. 10 shows a best common subgraph deter- 
mined by the graph matching section of the 
scene recognition portion of Fig. 2; and 
Figs. 11a and 11b show a reference scene and 

45 a sensed scene having the respective aimpoints 
designated therein. 

DETAILED DESCRIPTION 

so Referring to the drawings, Fig. 1 is a block 

diagram of a scene recognition system employing 
low and high level feature detection to identify and 
track targets in accordance with the principles of 
the present invention. The system comprises a low 

55 level feature detector 1 1 that is adapted to receive 
image data derived from an imaging sensor 9, such 
as an infrared sensor, a television sensor, or a 
radar, for example. The low level feature detector 
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11 is coupled to a high level graph matching pro- 
cessor 12 that is adapted to process icons repre- 
sentative of features contained in the image scene 
that are generated by the low level feature detector 
11. Aimpoint information is generated and used by 
a missile navigation system 24 to steer a missile 
toward a desired target. 

The high level graph matching processor 12 
includes a serially coupled graph synthesizer 13, 
graph matcher 14, and aimpoint estimator 15. A 
reference graph storage memory 16 is coupled to 
the graph matching processor 12 and is adapted to 
store predefined graphs representing expected tar- 
get objects that are present in the image scene. 
The reference graphs include graphs representing 
tanks, buildings, landmarks, and water bodies, for 
example. 

The low level feature detector 1 1 is described 
in detail in European patent application Serial No. 
91 105 250 filed on April 3, 1991, entitled 
"Improved Data Decompression System and Meth- 
od", whose teachings are incorporated herein by 
reference. In summary, however, the low level fea- 
ture detector 11 converts a matrix of sensor data 
(image data) into a matrix of orthogonal icons that 
symbolically represent the imaged scene through 
the use of a set of attributes. These are discussed 
in more detail below with reference to Figs. 3-1 1 . 

The processing performed by the scene rec- 
ognition system shown in Fig. 1 employs a series 
of transformations that converts image information 
into progressively more compressed and abstract 
forms. The first transformation performed by the 
low level feature detector 11 converts the sensor 
image, which is an array of numbers describing the 
intensity at each picture position, into a more com- 
pact array of icons with attributes that describe the 
essential intensity distributions of 10x10 pixel 
blocks of image data. 

The second transformation implemented in the 
high level graph matching processor 12 links the 
icon array into separate regions of nearly uniform 
intensity and identifies linear boundaries between 
regions. A special case of a region that is long and 
narrow with roughly parallel sides is identified as a 
ribbon. This transformation results in a list of the 
objects formed, various attributes of the objects 
themselves, and relationships between the objects. 
This information is encoded in a structure identified 
as an attributed graph. 

The graph matcher 14 compares the graph 
structure derived from the sensor image with a 
previously stored graph derived earlier or from 
reconnaissance information, for example. If a high 
degree of correlation is found, the scene described 
by the reference graph is declared to be the same 
as the scene imaged by the sensor. Given a 
match, the aimpoint estimator 15 associates the 



BNSOOCID: <EP 049251 2A2_I_> 



aimpoint given apriori in the reference graph to the 
sensed graph, and this information is provided as 
an input to the missile navigation and guidance 
system. 

5 In a classical graphing application, the general 

problem of correlating two graphs is N-P complete, 
in that a linear increase in the number of nodes 
requires an exponential increase in the search, or 
number of comparisons, required to match the 
10 graphs. In the present invention, this problem is 
solved by overloading the sensed and reference 
graphs with unique attributes that significantly re- 
duces the search space and permits rapid, real 
time searching to be achieved. 

15 Fig. 2 is a detailed functional block diagram of 

the system of Fig. 1. Scene segmentation process- 
ing 21 comprises low level feature description pro- 
cessing, intensity and gradient based segmentation 
processing 32, 33 and gradient/intensity object 

20 merging 34, all performed by the low level feature 
detector 11 (Fig. 1). Output signals from the scene 
segmentation processing 21 are coupled to scene 
description processing 22 that comprises graph 
synthesis processing 35 that produces a sensed 

25 graph of objects present in the image. Reference 
graphs 36 are stored that comprise graphs repre- 
senting vehicles, buildings, landmarks, and water 
bodies, for example. These graphs are prepared 
from video data gathered during reconnaissance 

30 flights, for example, which provide mission planning 
information. Target objects that are present in the 
video data are processed by the present invention 
to generate the reference graphs which are ulti- 
mately used as comparative data from which target 

35 objects are identified and selected during oper- 
ational use of the invention. 

Scene recognition processing 23 employs the 
graphs generated by the graph synthesis process- 
ing 35 and the reference graphs 36 by means of 

40 graph matching processing 37 and scene trans- 
formation processing 38 which generate the aim- 
point estimate that is fed to the navigation system 
24. 

Figs. 3a- 3f show the processing performed to 
45 achieve low level feature discrimination. This pro- 
cess converts a matrix of sensor data (image data) 
into a matrix of orthogonal icons that represent the 
imaged scene symbolically via a set of attributes. 
These orthogonal icons serve as the basis for per- 
50 forming symbolic scene segmentation, description, 
and recognition processors shown in Fig. 2. Fig. 3a 
represents an image of a house 41 whose sides 
42, 43 and roof sections 44, 45 have different 
shading due to the orientation of the sun relative 
55 thereto. Each face of the house 41 is identified by 
a different texture (shading) identified in Fig. 3a. 
The image of the house 41 is an array of numbers 
representative of different intensities associated 

4 
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with each pixel of data. 

The image data representing the house 41 is 
processed by a 10x10 block window 46 as is 
roughly illustrated in Fig. 3b which generates pixel 
data illustrated in Fig. 3c. The pixel data of Fig. 3c 
is generated in the low level feature detector 1 1 . 
This pixel data contains orthogonal icons. Each 
pixel in Fig. 3c is shaded in accordance with the 
shade it sees in Fig. 3b. The blocks or cells of data 
shown in Fig. 3c are generated in the low level 
feature detector 11. 

In Fig. 3d lines are formed, and in Fig. 3e 
regions are formed from the shaded pixels in Fig. 
3c. The lines are generated by gradient based 
segmentation processing 33 in the scene segmen- 
tation processor 21 shown in Fig. 2, while the 
regions are generated by the intensity based seg- 
mentation processing 32 in the scene segmentation 
processor 21 shown in Fig. 2. The regions are 
formed using flat linking and gradient linking and 
the region shapes are determined. The line and 
region information is then processed by 
gradient/intensity object merging 34 and the graph 
synthesis processing 35 shown in Fig. 2. in order 
to form regions and ribbons, and a graph is syn- 
thesized, as illustrated in Fig. 3f, which is a graph 
representation of the house 41 of Fig. 3a. An aim- 
point is shown in Fig. 3f that is derived from the 
information comprising the reference graph 36 
shown in Fig. 2. 

The flat linking and region growth processing is 
part of the scene segmentation processing 21 of 
Fig. 2. This flat linking and region growth process- 
ing is comprised of two subprocesses. The flat 
linking process groups low level feature discrimina- 
tion orthogonal icons of type FLAT into homo- 
geneous intensity flat regions using a relaxation- 
based algorithm. The result is a set of regions 
comprised of FLAT icons and described by their 
area (number of constituent FLAT icons), intensity 
(average intensity of the constituent FLAT icons), 
and a list of the constituent FLAT icons. More 
particularly, groups of orthogonal icons having 
homogeneous intensity regions are formed to gen- 
erate a set of regions having a block resolution 
boundary. The groups of orthogonal icons are com- 
prised of homogeneous intensity icons described 
by their area, including the number of constituent 
icons having homogeneous intensity, the average 
intensity of the constituent homogeneous intensity 
icons, and a list of the constituent homogeneous 
intensity icons, region growth processing appends 
adjacent orthogonal icons having an intensity gra- 
dient thereacross to provide a feature-resolution 
boundary. 

The region growth process appends adjacent 
orthogonal icons of type gradient (non FLAT) in- 
formation to the grown flat regions thus providing a 
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feature-resolution boundary as opposed to the 
10x10 block resolution boundary provided by the 
flat linking process. The method employed is as 
follows. (1) Begin with a flat region provided by the 

5 flat linking process. The region is comprised solely 
of FLAT icons and grows up to flanking gradient 
icons which stop the growth. (2) Consider the gra- 
dient icons flanking the region. Each gradient icon 
is described by a bi-intensity model of which one 

10 of the intensities are adjacent to the flat region. If 
this intensity is similar to the intensity of the flat 
region extend the region boundary to include the 
portion of the gradient icon covered by the inten- 
sity. (3) Repeat for all flat regions. Flat linking 

75 assigns a single region number to every FLAT 
icon. Region growth assigns multiple region num- 
bers to gradient icons. Gradient icons of type 
EDGE and CORNER are assigned two region num- 
bers, those of type RIBBON are assigned three 

20 region numbers. Gradient icons of type SPOT are 
not considered. The result of this process is a set 
of homogeneous intensity regions consisting of 
FLAT icons and partial gradient icons. This is de- 
scribed in detail in the "Improved Data Decompres- 

25 sion System and Method" patent application cited 
above. 

Figs. 4a and 4b show the results of flat linking 
and region growth processing performed by the 
scene segmentation processing 21 of Fig. 2. In 

30 particular this processing is performed by intensity 
based segmentation processing 32, gradient based 
segmentation processing 33, and the 
gradient/intensity object merging 34 processes 
shown in Fig. 2. In Figs. 4a and 4b, the orthogonal 

35 icons containing a single shade of gray represent 
FLAT icons 50, 51. Those that contain two shades 
of gray (along the diagonal) represent EDGE icons 
52. Regions are comprised of both FLAT and gra- 
dient (non FLAT) icons. EDGE icons 52, as well as 

40 all other gradient icons, are associated with mul- 
tiple regions, that is, they are assigned multiple 
region numbers. In Figs. 4a and 4b, each of the 
EDGE icons 52 are assigned two region numbers, 
1 and 2. This provides us with boundaries that are 

45 accurate to the gradient features detected by the 
low level feature discrimination process. 

Boundary formation and linear feature extrac- 
tion processing is performed by the scene seg- 
mentation processing of Fig. 2. This processing is 

50 comprised of three subprocesses. The gradient 
boundaries of each region 60-63 (Fig. 5a) are tra- 
versed forming a gradient chain around each re- 
gion. The gradient chains for each region are ana- 
lyzed for linear segments 64-68 (Fig. 5b) and hav- 

55 ing pointers inserted into the chains at the begin- 
ning and end of each linear segment 64-68. The 
linear segments 64-68 are analyzed for the pres- 
ence of adjacent segments related to two or more 

5 
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regions that form line segments. The results of 
these processes include boundary descriptions for 
each of the regions 60-63 formulated by the flat 
linking and region growth processes, and linear 
segments 70-73 (Fig. 5c) represented by length, 
orientation, and end point coordinates. 

Figs. 5a-5c show the results of boundary for- 
mation and linear feature extraction processing per- 
formed in the scene segmentation processing 21 of 
Fig. 2. Fig. 5a shows the regions 60-63 derived by 
the flat linking and region growth process. Fig. 5b 
shows the boundary descriptions of each of the 
regions 60-63 as derived by the boundary forma- 
tion process. Fig. 5c shows the line segments 70- 
73 derived by the linear feature extraction process. 

The object formation process is comprised of 
two subprocesses. The first forms ribbon objects 
75 from the linear features provided by the gradient 
boundary and linear feature extraction processes 
and creates symbolic descriptions of these objects. 
The second creates symbolic descriptions of the 
regions provided by the flat linking and region 
growing processes thus producing region objects 
76-78 (Figs. 8a, 8b). These ribbon and region ob- 
jects become the nodes of the attributed sensed 
graph. The symbolic descriptions serve as the at- 
tributes. Figs. 6 and 7 show attributes computed for 
region and ribbon objects 75-78, shown in Figs 8a 
and 8b, respectively, in object formation processing 
performed by the scene segmentation process 21 
of Fig. 2, and more particularly by the graph syn- 
thesis processing 35 in Fig. 2. 

Fig. 6 shows region object attributes and com- 
putations for a region boundary and a convex hull 
using the equations shown in the drawing. With 
reference to Fig. 7, it shows the ribbon object 
attributes, including ribbon length in Fig. 7a, ribbon 
intensity in Fig. 7b, polarity in Fig. 7c, and ribbon 
orientation in Fig. 7d. The arrows shown in Figs. 7b 
and 7c are indicative of the orientation of the line 
segments that constitute the ribbon. 

For region objects, the process computes the 
area, perimeter, and convex hull attributes. For 
ribbon objects, the process searches through a line 
table looking for pairs of lines that: (1) differ in 
orientation by 180 degrees, (2) are in close proxim- 
ity to each other, (3) are flanked by similar inten- 
sities, and (4) do not enclose another line segment 
that is parallel to either line. When a pair of lines 
fitting these constraints is found the ribbon at- 
tributes for them are computed. The attributes for 
ribbon objects include: (1) intensity of the ribbon, 
(2) polarity of the ribbon (light on dark or dark on 
light), (3) the width of the ribbon (distance between 
two lines), and (4) the orientation of the ribbon. 

The graph synthesis process and its asso- 
ciated graph language computes relational descrip- 
tions between the region and ribbon objects 75-78 
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(Figs. 8a, 8b) provided by the object formation 
process and formulates the attributed sensed graph 
structure from the region and ribbon objects and 
their relationships. The relationships between every 
5 region/region, region/ribbon, and ribbon/ribbon pair 
are computed. The relationships are of type spatial 
(proximity to one another) or comparative (attribute 
comparisons). Graphs, nodes, links between nodes, 
and graph formation and processing is well known 

70 in the art and will not be described in detail herein. 
One skilled in the art of graph processing should 
be able to derive the graphs described herein 
given a knowledge of the attributes that are to be 
referenced and the specification of the graphs de- 

15 scribed herein. 

In formulating the attributed sensed graph, the 
region and ribbon objects are placed at graph 
nodes, one object per node, along with their de- 
scriptive attributes. The relationships between each 

20 pair of objects are placed at the graph arcs (links) 
along with their attributes. With this scheme, a fully 
connected attributed graph is formulated which 
symbolically represents the original imaged scene. 
Figs. 8a-8c show the transitional steps in gen- 

25 erating an attributed sensed graph 80 in the graph 
synthesis process 35 (Fig. 2). The graph 80 is 
comprised of nodes 81-84, that symbolically repre- 
sent objects within the scene, and node attributes 
that describe the objects, and arcs (or links) that 

30 symbolically represent the relationships between 
the objects within the scene, and link attributes, 
that describe the relationships between the objects. 
More particularly, node 1 is representative of re- 
gion 1 object 76, node 2 is representative of region 

35 2 object 77, node 3 is representative of region 3 
object 78, and node 4 is representative of the 
ribbon object 75. The various links between the 
ribbon and region objects 75-78 are shown as 1 
link 2, 2 link 4, 1 link 4, 1 link 3, 2 link 3, and 3 link 

40 4. The links include such parameters as the fact 
that the region 1 object 76 is above and adjacent 
the region 2 object 77, that the region 1 object 76 
is left of the region 3 object 78, that the region 2 
object 77 is left and adjacent to the ribbon object 

45 75, and so forth. 

Reference graph processing entails the same 
functions as the object formation and graph synthe- 
sis processes with two differences. First, it is per- 
formed prior to the capture of the sensed image 

so and subsequent scene segmentation and descrip- 
tion. Second, it receives its inputs from a human 
operator by way of graphical input devices rather 
than from autonomous segmentation processes. 
Figs. 9a and 9b show a reference scene 90 

55 and an attributed reference graph 91, respectively. 
Fig 9a shows the reference scene 90 comprised of 
region 1 and region 2, the ribbon and a desired 
aimpoint 94. The attributed reference graph 91 

6 
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shown in Fig. 9b is comprised of nodes and links 
substantially the same as is shown in Fig. 8c. The 
nodes and links comprise the same types of rela- 
tionships described with reference to Fig. 8c. 

The graph matching process 37 {Fig. 2) com- 5 
pares attributed reference and attributed sensed 
graphs, generates a best common subgraph be- 
tween the two based on the degree of similarity 
between the two graphs (confidence number), and 
generates a recognition decision based on the val- 10 
ue of the confidence number and a predetermined 
threshold. Feasibility is determined by the degree 
of match between the node and arc attributes of 
each of the graphs. A heuristic procedure is in- 
cluded to ignore paths that cannot produce a con- 75 
fidence number larger than the predetermined 
threshold. Also, paths which lead to ambiguous 
solutions, that is, solutions that match a single 
object in one graph to multiple objects in the other, 
are ignored. Finally, paths that do not preserve the 20 
spatial relationships between the objects as they 
appeared in the original scene are ignored. 

Fig. 10 shows a best common subgraph 92 
determined by the graph matching section of the 
scene recognition processing 23 of Fig. 2. The 25 
graph shown in Fig. 8c is matched against the 
graph shown in Fig. 9b which generates the com- 
mon subgraph 92 of Fig. 10. This is performed in a 
conventional manner well-known in the graph pro- 
cessing art and will not be described in detail 30 
herein. 

Figs. 11a and 11b show a reference scene 90 
and a sensed scene 93 having their respective 
aimpoints 94, 95 designated therein. More particu- 
larly, Figs. 11a and 11b depict one possible see- 35 
nario in which an object is detected in the sensed 
scene (region 3) but is not designated in the refer- 
ence graph. An appropriate aimpoint is designated 
in spite of this difference as well as others includ- 
ing object additions or omissions, object size and 40 
shape differences, and changes due to imaging 
conditions. The aimpoint 94 in the reference scene 
90 is included in the attributed reference graph and 
is indicative of a desired target aiming location. 
This aimpoint is "transferred" to the sensed scene 45 
as the aimpoint that the missile should be directed 
at. This information is transferred to the missile 
navigation system for its use. 

Thus there has been described anew and im- 
proved scene recognition system and method that 50 
employs low and high level feature detection to 
identify and track targets. It is to be understood 
that the above-described embodiment is merely 
illustrative of some of the many specific embodi- 
ments which represent applications of the princi- 55 
pies of the present invention. Clearly, numerous 
and other arrangements can be readily devised by 
those skilled in the art without departing from the 
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scope of the invention. 
Claims 

1. A scene recognition system employing low 
and high level feature detection to identify and 
track targets located in an imaged scene (93) 
and a missile guidance system adapted to 
steer a missile toward a desired target, char- 
acterized by: 

- a low level feature detection processor 
(11; 31) adapted to process image data 
derived from and representative of the 
imaged scene (93), that extracts features 
from the imaged scene (93) by convert- 
ing the image data into a matrix of or- 
thogonal icons (50, 51) that symbolically 
represent the image using a predeter- 
mined set of attributes; 

- a graph synthesis processor (13; 35) 
coupled to the low level feature detection 
processor (11; 31) that processes the 
orthogonal icons (50, 51 ) to form objects 
(75-78) that are present in the imaged 
scene (93), and to form an attributed 
sensed graph (80) that is derived from 
the objects (75-78) and their relationships 
as described by their attributes, and 
whereupon each respective object (75- 
78) is placed at a graph node (81-84), 
along with its descriptive attributes, the 
relationships between object pairs are 
placed at graph links along with their 
attributes that link the nodes (81-84), 
whereupon a fully connected attributed 
graph (80) is formed that symbolically 
represents the image; 

- reference graph storage means (16; 36) 
coupled to the graph synthesis processor 
(13; 35) for storing predetermined refer- 
ence graphs (91) representative of iden- 
tifiable targets of interest that are ex- 
pected to be present in the image; and 

- a graph matching processor (14; 37) 
coupled to the graph synthesis processor 
(13; 35) that compares predetermined at- 
tributed reference graphs (91) to the at- 
tributed sensed graphs (80) to produce 
an object recognition decision based on 
the value of the degree of similarity be- 
tween the attributed reference graphs 
(91) and the attributed sensed graphs 
(80), and for providing an output signal 
that is determinative of a target aimpoint 
(94, 95), which output signal is coupled 
as an input to the missile guidance sys- 
tem to provide a guidance signal that is 
adapted to steer the missile toward the 
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identified target. 

2. A scene recognition system employing low 
and high level feature detection to identify and 
track targets located in an imaged scene (93) 5 
and a missile guidance system adapted to 
steer a missile toward a desired target, said 
system comprising: 

- low level feature detection processor 
means (11; 31) for processing image 10 
data derived from and representative of 

an imaged scene (93), and for extracting 
features from the imaged scene (93) by 
converting the image data into a matrix 
of orthogonal icons (50, 51) that symboli- is 
cally represent the image using a pre- 
determined set of attributes; 

- graph synthesis processor means (13; 
35) coupled to the low level feature de- 
tection processing means (11; 31) for 20 
processing the orthogonal icons (50, 51) 

to generate predetermined objects (75- 
78) representative of objects (75-78) that 
are in the scene, and for computing rela- 
tional descriptions between the objects 25 
(75-78) to form an attributed sensed 
graph (80) from the objects (75-78) and 
their relationships as described by their 
attributes, and whereupon the objects 
(75-78) are placed at graph nodes (81- 30 
84), one object per node (81-84), along 
with their descriptive attributes, and 
wherein the relationships between object 
pairs are placed at graph links along with 
their attributes, and whereupon a fully 35 
connected attributed graph (80) is formu- 
lated which symbolically represents the 
imaged scene (93); 

- reference graph storage means (13; 36) 
coupled to the graph synthesis process- 40 
ing means (13; 35) for storing predeter- 
mined reference graphs (91) representa- 
tive of identifiable targets of interest that 

are expected to be present in the data 
comprising the image; and 45 

- graph matching processing means (14; 
37) coupled to the graph synthesis pro- 
cessing means (13; 35) for comparing 
predetermined attributed reference 
graphs (91) to the sensed graphs (80) to 50 
produce an object recognition decision 
based on the value of the degree of 
similarity between the attributed refer- 
ence graphs (91) to the sensed graphs 

(80) and a predetermined threshold, and 55 
for providing an output signal that is de- 
terminative of a target aimpoint (94, 95), 
which output signal is coupled as an 



input to the missile guidance system to 
provide a guidance signal that is adapted 
to steer the missile toward the identified 
target. 

3. The system of claim 1 or 2, characterized in 
that the low level feature detection processing 
means (11; 31) comprises: 

- flat linking processing means coupled to 
the low level feature detection processing 
means (11; 31) for forming groups of 
orthogonal icons (50, 51) having homo- 
geneous intensity regions (60-63) to gen- 
erate a set of regions having a block 
resolution boundary and that are com- 
prised of homogeneous intensity icons 
described by their area, comprising the 
number of constituent icons having 
homogeneous intensity, the intensity, 
comprising the average intensity of the 
constituent homogeneous intensity icons, 
and a list of the constituent homoge- 
neous intensity icons; and 

- region growth processing means coupled 
to the flat linking processing means for 
appending adjacent orthogonal icons (50, 
51) having an intensity gradient 
thereacross to provide a feature-resolu- 
tion boundary (64-68). 

4. The system of claim 3, characterized in that 
the low level feature detection processing 
means (1 1 ; 31 ) further comprises: 

- boundary formation and linear feature ex- 
traction processing means coupled to the 
region growth processing means for 

- traversing the gradient boundaries 
(64-68) of each region to form a gra- 
dient chain around each region, 

- analyzing the gradient chains for each 
region (60-63) for linear segments (70- 
73) by means of pointers inserted into 
the chains at the beginning and end of 
each linear segment (70-73), and 

- analyzing the linear segments (70-73) 
for the joining of segments (70-73) 
related to two or more regions (60- 
63) 

to form linear segments (70-73), which 
generates boundary descriptions for each 
of the regions (60-63) formulated by the 
flat linking and region growth processing 
means and linear segments (70-73) re- 
presented by length, orientation, and end 
point coordinates. 

5. The system of claim 4, characterized in that 
the low level feature detection processing 
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means (11; 31) further comprises: 

- object formation processing means coup- 
led to the boundary formation and linear 
feature extraction processing means for 
forming ribbon objects (75) from the tin- 5 
ear features provided by the gradient 
boundary and linear feature extraction 
processing means and for creating sym- 
bolic descriptions of these objects, by 
creating symbolic descriptions of the re- 10 
gions (60-63) provided by the flat linking 

and region growing processing means to 
produce region objects (76-78) that de- 
fine nodes (81-84) of an attributed 
sensed graph (80), and whose symbolic 75 
descriptions comprise the attributes; 

- whereby for region objects (76-78), the 
object formation processing means com- 
putes the area, perimeter, and convex 

hull attributes, and for the ribbon objects 20 
(75), the object formation processing 
means searches through a line table 
looking for pairs of lines that: 

- differ in orientation by 180 degrees, 

- are in close proximity to each other, 25 

- are flanked by similar intensities, 

- do not enclose another line segment 
that is parallel to either line, 

and when a pair of lines fitting these 
constraints is found, the ribbon attributes 30 
for them are computed, and wherein the 
attributes for ribbon objects (75) include: 

- intensity of the ribbon (75), 

- polarity of the ribbon (75), meaning 

light on dark or dark on light, 35 

- the width of the ribbon (75), meaning 
the distance between the two lines, 
and 

- the orientation of the ribbon (75). 

40 

6. The system of any of claims 1 through 5, 
characterized in that the graph matching pro- 
cessing means (14; 37) compares predermined 
attributed reference graphs (91) to the sensed 
graphs (80) to produce a best common subg- 45 
raph (92) between the two based on the de- 
gree of similarity between the two graphs (80, 

91), and generates a recognition decision 
based on the value of the degree of similarity 
and a predetermined threshold. 50 

7. The system of claim 6, characterized in that 
the graph matching processing means (14; 37) 
utilizes a heuristically directed depth-first 
search technique to evaluate feasible matches 55 
between nodes and arc attributes of the attrib- 
uted reference and sensed graphs (80, 91). 



8. The system of claim 6 or 7, characterized in 
that the graph matching processing means (14; 
37) determines feasibility by the degree of 
match between the node and arc attributes of 
each of the graphs (80, 91), and a heuristic 
procedure is included to ignore paths that can- 
not produce a degree of similarity larger than 
the predetermined threshold. 

9. The system of any of claims 6 - 8, character- 
ized in that the paths which lead to ambiguous 
solutions, comprising solutions that match a 
single object in one graph (80, 91) to multiple 
objects in the other, are ignored, and wherein 
paths that do not preserve the spatial relation- 
ships between the objects as they appeared in 
the image are ignored. 

10. A scene recognition system employing low 
and high level feature detection to identify and 
track targets located in an imaged scene (93) 
and a missile guidance system adapted to 
steer the missile toward a desired target, char- 
acterized by: 

- low level feature detection processing 
means (11; 31) adapted to process im- 
age data derived from and representative 
of an imaged scene (93), for extracting 
low level features from the image (93) by 
converting the image data into a matrix 
of orthogonal icons (50, 51) that symboli- 
cally represent the imaged scene (93) 
using a predetermined set of attributes; 

- graph synthesis processing means (13; 
35) coupled to the low level feature de- 
tection processing means (11; 31) pro- 
cessing the orthogonal icons (50, 51) and 
for computing relational descriptions be- 
tween the region (76-78) and ribbon ob- 
jects (75) provided by the object forma- 
tion processing means to form the attrib- 
uted sensed graph (80) from the region 
and ribbon objects (75-78) and their rela- 
tionships, and whereupon the region and 
ribbon objects (75-78) are placed at 
graph nodes (81-84), one object per 
node (81-84), along with their descriptive 
attributes, the relationships between each 
pair of objects (75-78) are placed at the 
graph links along with their attributes, 
whereupon, a fully connected attributed 
graph (80) is formulated which symboli- 
cally represents the image (93); 

- reference graph storage means (16; 36) 
coupled to the graph synthesis process- 
ing means (13; 35) for storing predeter- 
mined reference graphs (91) representa- 
tive of identifiable targets of interest that 
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are expected to be present in the data 
comprising the image; and 

- graph matching processing means (14; 
37) coupled to the graph synthesis pro- 
cessing means (13; 35) for comparing 5 
predetermined attributed reference 
graphs (91) to the sensed graphs (80) to 
produce an object recognition decision, 
which produces a best common subg- 
raph (92) between the two based on the io 
degree of similarity between the two 
graphs (80, 91), for generating a recogni- 
tion decision based on the value of the 
degree of similarity and a predetermined 
threshold, and for providing an output 75 
signal that is determinative of a target 
aimpoint (94, 95), which output signal is 
coupled as an input to the missile guid- 
ance system to provide a guidance sig- 
nal that is adapted to steer the missile 20 
toward the identified target. 

11. A scene recognition system employing low 
and high level feature detection to identify and 
track targets located in an imaged scene (93), 25 
characterized by: 

- low level feature detection processing 
means (11; 31) adapted to process im- 
age data derived from and representative 

of an imaged scene (93), for extracting 30 
low level features from the imaged scene 
(93) by converting the image data into a 
matrix of orthogonal icons (50, 51) that 
symbolically represents the image (93) 
using a predetermined set of attributes, 35 

- flat linking processing means coupled to 
the low level feature detection processing 
means (11; 31) for forming groups of 
orthogonal icons (50, 51) having homo- 
geneous intensity regions (60-63) by 40 
means of a relaxation-based algorithm to 
generate a set of regions (60-63) having 

a block resolution boundary and that are 
comprised of homogeneous intensity 
icons described by their area, comprising 45 
the number of constituent icons having 
homogeneous intensity, the intensity, 
comprising the average intensity of the 
constituent homogeneous intensity icons 
(50, 51), and a list of the constituent 50 
homogeneous intensity icons (50, 51); 

- region growth processing means coupled 
to the fiat linking processing means for 
appending adjacent orthogonal icons (50, 

51) having an intensity gradient 55 
thereacross to provide a feature-resolu- 
tion boundary (64-68); 

- boundary formation and linear feature ex- 



traction processing means coupled to the 
region growth processing means for 

- traversing the gradient boundaries 
(64-68) of each region to form a gra- 
dient chain around each region, 

- analyzing the gradient chains for each 
region (60-63) for linear segments (70- 
73) by means of pointers inserted into 
the chains at the beginning and end of 
each linear segment (70-73), and 

- analyzing the linear segments (70-73) 
for the joining of segments (70-73) 
related to two or more regions (60- 
63) 

to form linear segments (70-73), which 
generates boundary descriptions for each 
of the regions (60-63) formulated by the 
flat linking and region growth processing 
means and linear segments (70-73) re- 
presented by length, orientation, and end 
point coordinates; 

- object formation processing means coup- 
led to the boundary formation and linear 
feature extraction processing means for 
forming ribbon objects (75) from the lin- 
ear features provided by the gradient 
boundary and linear feature extraction 
processing means and for creating sym- 
bolic descriptions of these objects, by 
creating symbolic descriptions of the re- 
gions (60-63) provided by the flat linking 
and region growing processing means to 
produce region objects (76-78) that de- 
fine nodes (81-84) of an attributed 
sensed graph (80), and whose symbolic 
descriptions comprise the attributes; 

- whereby for region objects (76-78), the 
object formation processing means com- 
putes the area, perimeter, and convex 
hull attributes, and for the ribbon objects 
(75), the object formation processing 
means searches through a line table 
looking for pairs of lines that: 

- differ in orientation by 180 degrees, 

- are in close proximity to each other, 

- are flanked by similar intensities, 

- do not enclose another line segment 
that is parallel to either line, and 

when a pair of lines fitting these con- 
straints is found, the ribbon attributes for 
them are computed, and wherein the at- 
tributes for ribbon objects (75) include: 

- intensity of the ribbon (75), 

- polarity of the ribbon (75), meaning 
light on dark or dark on light, 

- the width of the ribbon (75), meaning 
the distance between the two lines, 
and 
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- the orientation of the ribbons (75); 

- graph synthesis processing means (13; 
35) coupled to the low level feature de- 
tection processing means (11; 31) pro- 
cessing the orthogonal icons (50, 51) and 5 
for computing relational descriptions be- 
tween the region and ribbon objects (75- 

78) provided by the object formation pro- 
cessing means to form the attributed 
sensed graph (80) from the region and io 
ribbon objects (75-78) and their relation- 
ships, and whereupon the region and rib- 
bon objects (75-78) are placed at graph 
nodes (81-84), one object per node (81- 
84), along with their descriptive at- 75 
tributes, the relationships between each 
pair of objects (75-78) are placed at the 
graph links along with their attributes, 
whereupon, a fully connected attributed 
graph (80) is formulated which symboli- 20 
cally represents the imaged scene (93); 

- reference graph storage means (16; 36) 
coupled to the graph synthesis process- 
ing means (13; 35) for storing predeter- 
mined reference graphs (91) representa- 25 
tive of identifiable targets of interest that 

are expected to be present in the data 
comprising the imaged scene (93); and 

- graph matching processing means (14; 

37) coupled to the graph synthesis pro- 30 
cessing means (13; 35) for comparing 
predetermined attributed reference 
graphs (91) to the sensed graphs (80) to 
produce a best common subgraph (92) 
between the two (80, 91) based on the 35 
degree of similarity between the two 
graphs (80, 91), and for generating a 
recognition decision based on the value 
of the degree of similarity and a pre- 
determined threshold, and wherein the 40 
graph matching processing means (14; 
37) utilizes a heuristically directed depth- 
first search technique to evaluate feasible 
matches between the nodes (81-84) and 
arcs of the attributed reference and 45 
sensed graphs (80, 91), wherein feasibil- 
ity is determined by the degree of match 
between the node and arc attributes of 
each of the graphs (80, 91 ), and a heuris- 
tic procedure is included to ignore paths so 
of the tree that cannot possibly produce 
a degree of similarity larger than the 
predetermined threshold, and wherein 
the paths of the tree which lead to am- 
biguous solutions, wherein solutions that 55 
match a single object in one graph (80, 
91) to multiple objects in the other, are 
ignored, and wherein paths of the tree 



that do not preserve the spatial relation- 
ships between the objects (75-78) as 
they appeared in the original scene (93) 
are ignored. 

12. A method for use with a missile guidance 
system to track targets located in an imaged 
scene (93), characterized by the steps of: 

- processing image data derived from and 
representative of an imaged scene (93), 
and extracting low level features from the 
image (93) by converting the image data 
into a matrix of orthogonal icons (50, 51) 
that symbolically represent the imaged 
scene (93) using a predetermined set of 
attributes; 

- processing the orthogonal icons (50, 51) 
to form an attributed sensed graph (80) 
from region and ribbon objects (75-78) 
comprising the orthogonal icons (50, 51) 
and their relationships, and whereupon 
the region and ribbon objects (75-78) are 
placed at graph nodes (81-84), one ob- 
ject per node (81-84), along with their 
descriptive attributes, the relationships 
between each pair of objects (75-78) are 
placed at the graph links along with their 
attributes, whereupon, a fully connected 
attributed graph (80) is formulated which 
symbolically represents the imaged 
scene (93); 

- storing predetermined reference graphs 
(91) representative of identifiable targets 
of interest that are expected to be 
present in the data comprising the im- 
aged scene (93); and 

- comparing predetermined reference 
graphs (91) to the attributed sensed 
graphs (80) to produce an object rec- 
ognition decision, and providing an out- 
put signal that is determinative of a tar- 
get aimpoint (94, 95), which output signal 
is coupled as an input to the missile 
guidance system to provide a guidance 
signal that is adapted to steer a missile 
toward the identified target. 

13. The method of claim 12, characterized in that 
the step of processing image data comprises: 

- forming groups of orthogonal icons (50, 
51) having homogeneous intensity re- 
gions by means of a relaxation-based 
algorithm to generate a set of regions 
(60-63) having a block resolution bound- 
ary and that are comprised of homo- 
geneous intensity icons (50, 51) de- 
scribed by their area, comprising the 
number of constituent icons (50, 51) hav- 
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ing homogeneous intensity, the intensity, 
comprising the average intensity of the 
constituent homogeneous intensity icons 
(50, 51), and a list of the constituent 
homogeneous intensity icons (50, 51); 
and 

- appending adjacent orthogonal icons (50, 
51) having an intensity gradient 
thereacross to provide a feature-resolu- 
tion boundary (64-68). 

14. The method of claim 12 or 13, characterized in 
that the step of processing image data com- 
prises: 

- forming regions (60-63) by 

- traversing the gradient boundaries 
(64-68) of each region (60-63) to form 
a gradient chain around each region, 

- analyzing the gradient chains for each 
region (60-63) for linear segments (70- 
73) by means of pointers inserted into 
the chains at the beginning and end of 
each linear segment (70-73), and 

- analyzing the linear segments (70-73) 
for the joining of segments (70-73) 
related to two or more regions (60- 
63) 

to form linear segments (70-73), which 
generates boundary descriptions (64-68) 
for each of the regions (60-63) formu- 
lated by the flat linking and region growth 
processing means and linear segments 
(70-73) represented by length, orienta- 
tion, and end point coordinates. 

15. The method of claim 14, characterized in that 
the step of processing image data comprises: 

- forming ribbon objects (75) from the lin- 
ear features provided by the gradient 
boundary and linear feature extraction 
processing step and for creating sym- 
bolic descriptions of these objects (75), 
by creating symbolic descriptions of the 
regions (75) provided by the flat linking 
and region growing processing means to 
produce region objects (75-78) that de- 
fine nodes (81 -84) of an attributed 
sensed graph (80), and whose symbolic 
descriptions comprise the attributes; 

- whereby for region objects (75-78), the 
area, perimeter, and convex hull at- 
tributes are determined, and for the rib- 
bon objects (75), a line table is searched 
looking for pairs of lines that: 

- differ in orientation by 180 degrees, 

- are in close proximity to each other, 

- are flanked by similar intensities, and 

- do not enclose another line segment that 
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is parallel to either line, 
and when a pair of lines fitting these con- 
straints is found, the ribbon attributes for them 
are computed, and wherein the attributes for 
ribbon objects (75) include: 

- intensity of the ribbon (75), 

- polarity of the ribbon (75), meaning light 
on dark or dark on light, 

- the width of the ribbon (75), meaning the 
distance between the two lines, and 

- the orientation of the ribbon (75). 



16. The method of any of claims 12 - 15, char- 
acterized in that the comparing step comprises 

75 comparing predetermined reference graphs 

(91) to the attributed sensed graphs (80) based 
on the value that is a function of the difference 
between the degree of similarity between the 
reference and sensed graphs (80, 91), and a 

20 predetermined threshold. 

17. The method of claim 16, characterized in that 
the comparing step utilizes a heuristically di- 
rected depth-first search technique to evaluate 

25 feasible matches between nodes (81-84) and 

arcs of the attributed reference and sensed 
graphs (80, 91). 

18. The method of claim 16 or 17, characterized in 
30 that the comparing step determines feasibility 

by the degree of match between the node and 
arc attributes of each of the graphs (80, 91), 
and a heuristic procedure is included to ignore 
paths that cannot produce a degree of similar- 
35 ity larger than the predetermined threshold. 

19. The method of any of claims 16-18, char- 
acterized in that the paths which lead to am- 
biguous solutions, comprising solutions that 

40 match a single object in one graph to multiple 

objects in the other, are ignored, and wherein 
paths that do not preserve the spatial relation- 
ships between the objects as they appeared in 
the imaged scene (93) are ignored. 

45 

20. A method of identifying a predefined object 
located in an imaged scene (93), characterized 
by the steps of: 

- storing representations of predefined ob- 
50 jects in a predetermined graph format; 

- processing image data to generate icons 
(50, 51) representative of image features 
contained in the imaged scene (93); 

- processing the icons (50, 51) to form 
55 objects (75-78) contained in the imaged 

scene (93); 

- synthesizing objects (75-78) into repre- 
sentations of objects in the imaged 
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scene (93); 

- comparing the synthesized representa- 
tions of objects (75-78) in the imaged 
scene (93) to the stored representations 

of predefined objects to determine the 5 
existence of predefined associations 
therebetween; and 

- providing an output signal indicative of a 
match between a synthesized represen- 
tation and a stored representation, which io 
output signal is indicative of the iden- 
tification of a predefined object located in 

the imaged scene (93). 

21. The method of claim 20, characterized in that 75 
the step of processing the icons (50, 51) to 
form objects (75-78) comprises the step of 
processing the icons (50, 51) using syntactic 
techniques on each icon (50, 51). 

20 

22. The method of claim 20 or 21, characterized in 
that the step of processing the icons (50, 51) 
to form objects (75-78) comprises the step of 
processing the icons (50, 51) using perceptual 
grouping techniques on each icon (50, 51). 25 
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© Scene recognition system and method employing low and high level feature processing. 



© A scene recognition system and a method em- 
ploying low and high level feature processing are 
disclosed. Image data is processed by a low level 
feature detection processor (31) that extracts low 
£J level features from an image. This is accomplished 
" by converting a matrix of image data into a matrix of 
^ orthogonal icons that symbolically represent the im- 
t™ age scene using a predetermined set of attributes. 

The orthogonal icons serve as the basis of process- 
^| ing by means of a high level graph matching proces- 
0£ sor (37) which employs symbolic scene segmenta- 
tion, description, and recognition processing that is 
O performed subsequent to the low level feature detec- 
n tion. This processing generates attributed graphs 
LU 



representative of target objects present in the image 
scene. High level graph matching compares pre- 
determined attributed reference graphs (36) to the 
sensed graphs (35) to produce a best common sub- 
graph between the two (35, 36) based on the degree 
of similarity between the two graphs (35, 36). The 
high level graph matching (37) generates a recogni- 
tion decision based on the value of the degree of 
similarity and a predetermined threshold. The output 
of the high level graph matching (37) provides data 
from which a target aimpoint is determined, and this 
aimpoint is coupled as an input to a missile guid- 
ance system (24) that tracks identified targets. 



BNSDOCID: <EP 049251 2 A3 J _> 



Rank Xerox (UK) Business Services 

O.08/3. 6/3.3. It 



EP 0 492 512 A3 



Image - 



Low level 
Feature 
Description 



31 



Intensity Based 
Segmentation 



-32 



Scene Segmentation 



21 



Gradient Based 
Segmentation 



Gradient/Intensity 
Object Merging 

r? — 



34- 



-33 



Graph Synthesis 
(Sensed Graph) 



22 



,35 



36 



Reference Graph 



Graph Matching 



Scene Recognition 



10 



Fig. 2 



37 



Scene 
Transformation 

TP — 



38- 



Navigation System 



Scene Description 



23 



24 



BNSDOCID: <EP 049251 2A3_I_> 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Nmb> 

EP 91 12 1973 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant pa 



EP-A-0 388 210 (BRITISH AEROSPACE PLC.) 

* abstract; claim 1 * 

US-A-4 783 829 (MIYAKAWA ET AL.) 

* the whole document * 

EP-A-0 307 120 (THE GENERAL ELECTRIC 
COMPANY, P.L.C.) 

* the whole document * 

US-A-4 183 013 (AGRAWALA ET AL.) 

* the whole document * 



Rrlr I'lit 
to 



CLASSIFICATION OP THE 
APPLICATION On!. CM ) 



G06F15/66 
F41G7/22 
G06K9/62 
G01S3/786 



TECHNICAL FIELDS 
SEARCHED (1st. CLS ) 



G01S 
F41G 
G06K 



The present search report hat been drawn up for aJI 



BERLIN 



Dae ml fawiltlw «f U» i 

29 SEPTEMBER 1993 



OANIELIDIS S. 



CATEGORY OF CITED DOCUMENTS 

X : particularly relevant If takes aim 

Y : particularly relevant If eombtneri with another 

eoauaent of the same category 
A : techaotogjcal bavaground 
O : non-written disclosure 
P : Intermediate document 



T : theory or prfaidpJe underlying the Invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document dted In the application 
L : document dted for other reasons 

* : Bomber of the same patent fanUh/. corresponding* 
eocuaeot 



BNSDOCID: <EP 049251 2A3J > 



This Page Blank (uspto) 



